In the area of high speed data acquisition there has been a tradition of producing custom computing hardware solutions for recording bursts of data into buffer memory (for short bursts at high data rate) and custom storage solutions for storing this data to disk subsystems. In many cases impressive speeds are achieved for short bursts, but continuous operation at high data rate has been lacking Other systems such as the Conduant Mark 5 series have a custom hardware design connected to a special purpose disk array designed for continuous average rate recording.
Most solutions have been hardware centric with limited flexibility, large size and very costly to develop. Today it is possible to use software centric maintainable solution to high speed acquisition and data storage, by solving a limited set of challenges.
In order to efficiently collect large amounts of high speed data multiple storage devices in parallel are needed. Today these devices are computer hard drives with rotary or solid state device (SSD) media. The common theme is that there will be multiple such devices needed in order to store the data either limited by volume or by data rate. A fast rotary media disk can sustain data rates in the order of approximately 100-150 MB/s while a SSD can sustain 200-400 MB/s. With high data rates of several GB/s multiple drives in parallel are employed.
When recording large sets of data in the field, data needs to be shipped from the location of collection to some other location for processing. Common recording systems on the market, for example as shown in FIG. 1, use an internal RAID-based disk array 110 (often hot swappable for ease of maintenance) attached directly to the recording unit chassis 100 of the computer and collection equipment. According to the prior art embodiments, when disks needs to be replaced or maintained, the disks 111, 112, 113, 114, 115, 116, 117, 118, 119, 120 either need to be removed individually from the chassis (and put back in the chassis at the receiving site in the same order) or the entire chassis need to be shipped. Shipping the entire chassis in most cases becomes prohibitively expensive. More particularly, because RAID controllers are notoriously particular with respect to hardware and operating system version, any receiving site typically requires the exact same instantiation of the RAID controller/operating system. This can be problematic, especially if one of the computers has undergone an update to its hardware of (more typically) software.
Furthermore, when streaming data at high data rate to disks, the traditional solution has been to use a RAID (redundant array of independent disks) subsystem to group multiple disks together and increase the total bandwidth. Current commercial RAID systems can theoretically, on paper, achieve disk speeds up to approximately one GByte per second of transfer rate. Then multiple such RAID subsystems can potentially be run in parallel to achieve multi-GByte per second transfer rates. This is a common practice however there are at least two major drawbacks with this approach.
The first problem is that if a disk within a raid set has a problem (for example a write error, timeout, or it is slower than the others) the entire raid set slows down until the problem is corrected. If a disk breaks down the raid slows down into a fall back mode and thus destroys the performance of the whole set. In lab tests a 1 GByte/s Raid slow down has been observed to a few hundred MByte/s when there has been a fault on one out of 16 drives (disks) in a set.
The second problem is that moving the disks from one system to another is difficult to do. The file system on the disks is vendor, card, and version on the card firmware specific. Accordingly, the controllers on all machines that are able to read the data from a specific disk pack must be identical. This requirement sets severe limitations on portability, and possibilities for upgrading systems across multiple sites.
There are several challenges when performing high-speed recording of continuous data streams. It is generally desirable to provide a high speed data recorder that is reliable, resilient, cost effective and maintainable, as well as providing continuous operation with minimal human intervention.